Description

mvcombine lowered to Calcite ARRAY_AGG(<field>) FILTER (<field> IS NOT NULL), which the analytics-engine (DataFusion) route cannot execute: ARRAY_AGG is not in the engine's aggregate function set, so the plan is rejected during marking. Mapping it into the engine is not viable either — Calcite ARRAY_AGG infers a nullable-element ARRAY while the engine's array_agg infers a NOT NULL element, and the two cannot be reconciled during the aggregate rewrite.

The fix stays on the SQL side: performArrayAggAggregation now emits the existing PPL list() aggregate (PPLBuiltinOperators.LIST), which the analytics engine already supports end to end (it maps to the engine's array_agg / list_merge operators). ListAggFunction filters nulls internally, so the explicit IS NOT NULL filter is dropped. No analytics-engine (core) change is required.

Behavior change (please review): the combined field is now ARRAY<VARCHAR> — list() renders elements as strings and retains at most 100 values per group, per its documented contract. For string target fields (the documented mvcombine use) output is unchanged; numeric/other targets are now stringified and capped. On the Calcite/Lucene path this differs from the previous unbounded, type-preserving ARRAY_AGG, so flagging in case reviewers prefer to gate it.

Bug reproduce (analytics-engine / DataFusion route)

Composite/parquet index, analytics.planner.prefer_metadata_driver=false (forces DataFusion):

source=test_data | fields category, status | mvcombine status

Before (ARRAY_AGG) — rejected on the analytics engine

Logical plan:

LogicalProject(category=[$0], status=[$1])
  LogicalAggregate(group=[{0}], status=[ARRAY_AGG($1) FILTER $2])
    LogicalProject(category=[$0], status=[$7], $f2=[IS NOT NULL($7)])
      LogicalTableScan(table=[[opensearch, test_data]])

Physical plan: none — rejected during analytics-engine marking, before execution.

Output:

{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "No enum constant org.opensearch.analytics.spi.AggregateFunction.ARRAY_AGG",
    "type": "IllegalArgumentException"
  },
  "status": 500
}

After (LIST) — runs on DataFusion

Logical plan:

LogicalProject(category=[$0], status=[$1])
  LogicalAggregate(group=[{0}], status=[LIST($1)])
    LogicalProject(category=[$0], status=[$7])
      LogicalTableScan(table=[[opensearch, test_data]])

Physical plan (profile=true; every node viableBackends=[[datafusion]], stage chosen_backend=datafusion):

OpenSearchSort(fetch=[10000], viableBackends=[[datafusion]])
  OpenSearchProject(category=[$0], status=[$1], viableBackends=[[datafusion]])
    OpenSearchAggregate(group=[{0}], status=[LIST($1)], mode=[SINGLE], viableBackends=[[datafusion]])
      OpenSearchProject(category=[$0], status=[$7], viableBackends=[[datafusion]])
        OpenSearchTableScan(table=[[test_data]], viableBackends=[[datafusion]])

Output:

+----------+--------------------------+
| category | status                   |
|----------+--------------------------|
| A        | ["active", "active"]     |
| B        | ["inactive", "active"]   |
| C        | ["pending"]              |
+----------+--------------------------+

Related Issues

Related to #4766 ([FEATURE] Add mvcombine command in PPL) — extends it to the analytics-engine (DataFusion) route.

Check List

• New functionality includes testing.
  • CalcitePPLMvCombineTest logical-plan + Spark SQL expectations updated for the LIST aggregate. CalciteMvCombineCommandIT is unchanged (it already asserts string-array output on a string field).
• New functionality has been documented.
  • docs/user/ppl/cmd/mvcombine.md gains a Limitations note (string elements, 100-value cap, unordered).
• Commits are signed per the DCO using --signoff.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.